<html>
<head><meta charset="utf-8"><title>preventing stack overflows · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html">preventing stack overflows</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="243513752"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243513752" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243513752">(Jun 22 2021 at 13:15)</a>:</h4>
<p>I'm looking into <a href="https://github.com/rust-analyzer/rust-analyzer/issues/9358">https://github.com/rust-analyzer/rust-analyzer/issues/9358</a>, and it looks like rust-analyzer is generally a bit prone to run into stack overflows, which makes we wonder, perhaps we should make use of <code>stacker</code>, like rustc does, to prevent these problems from bringing down the whole server?</p>



<a name="243513858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243513858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243513858">(Jun 22 2021 at 13:15)</a>:</h4>
<p>Specifically, I managed to provoke a stack overflow in the parser, with macro output that is still well below the token limit of 524288 tokens</p>



<a name="243513990"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243513990" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243513990">(Jun 22 2021 at 13:16)</a>:</h4>
<p>Or maybe I'm overthinking this, and we should instead try to detect exponential macros once the output goes beyond a certain size?</p>



<a name="243514131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243514131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243514131">(Jun 22 2021 at 13:17)</a>:</h4>
<p>In this case we go from <code>input 36859 -&gt; output 73727</code> tokens just before the stack overflow</p>



<a name="243514253"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243514253" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243514253">(Jun 22 2021 at 13:18)</a>:</h4>
<p>I think we should rather place limits on the depths of things, and err out when we hit them. Stack space is just one resource, it's possible to burn loads of CPU without burning loads of stack.  See, for example, delightful <code>test_macro_expand_will_stop_2</code> test</p>



<a name="243514306"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243514306" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243514306">(Jun 22 2021 at 13:18)</a>:</h4>
<p>That being said, I think it's OK to bump the actual stack limit to 64 megs or something</p>



<a name="243514497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243514497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243514497">(Jun 22 2021 at 13:20)</a>:</h4>
<p>Ah, good call, I figured we'd have some related test</p>



<a name="243514528"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243514528" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243514528">(Jun 22 2021 at 13:20)</a>:</h4>
<p>For parser, we already have <code>start_node</code> / <code>finish_node</code>, so we can use those as depth check.</p>



<a name="243514642"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243514642" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243514642">(Jun 22 2021 at 13:21)</a>:</h4>
<p>The question is, what to do when we hit the bottom? I'd rather not return an <code>Error</code>, we should never fail. I guess, the parser can just give up and put all the rest of the tokens into an error node?</p>



<a name="243514892"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243514892" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243514892">(Jun 22 2021 at 13:23)</a>:</h4>
<p>Btw, in the <code>test_macro_expand_will_stop_2</code>'s <code>do_resolve</code>, try renaming <code>not_ra_fixture</code> to <code>ra_fixture</code> ^^</p>



<a name="243515206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243515206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243515206">(Jun 22 2021 at 13:25)</a>:</h4>
<p>no thanks :D</p>



<a name="243515272"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243515272" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243515272">(Jun 22 2021 at 13:26)</a>:</h4>
<p>I already crashed r-a several times while adding a test for the stack overflow</p>



<a name="243517038"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243517038" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243517038">(Jun 22 2021 at 13:38)</a>:</h4>
<p>Originally, it was names something like <code>code</code>, and I got into the trap of "fixing" it a couple of times</p>



<a name="243530394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243530394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243530394">(Jun 22 2021 at 15:05)</a>:</h4>
<p>hmm, this all seems to relate to how we recover from mismatched delimiters</p>



<a name="243530448"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243530448" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243530448">(Jun 22 2021 at 15:06)</a>:</h4>
<p>the macro only produces a subtree with a depth of &lt;30, but it contains a lot of unmatched <code>(</code> tokens that don't increase the depth, and the parser chokes on that</p>



<a name="243530540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243530540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243530540">(Jun 22 2021 at 15:06)</a>:</h4>
<p>perhaps we should change subtree lowering to always produce subtrees with matched delimiters?</p>



<a name="243531726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243531726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243531726">(Jun 22 2021 at 15:15)</a>:</h4>
<p>cc <a href="https://github.com/rust-analyzer/rust-analyzer/pull/4680#issuecomment-641346748">https://github.com/rust-analyzer/rust-analyzer/pull/4680#issuecomment-641346748</a></p>



<a name="243531760"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243531760" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243531760">(Jun 22 2021 at 15:15)</a>:</h4>
<p>hard problem, we need to figure out which invariants we want to have here</p>



<a name="243533131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243533131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243533131">(Jun 22 2021 at 15:23)</a>:</h4>
<p>I was thinking to just do it in the <code>syntax_bridge</code> without involving the parser</p>



<a name="243533150"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/preventing%20stack%20overflows/near/243533150" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/preventing.20stack.20overflows.html#243533150">(Jun 22 2021 at 15:23)</a>:</h4>
<p>would result in tokens that don't actually exist in the input though</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>